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ABSTRACT 



This evaluation attempts to measure the extent and 
effectiveness of ESEA Title I programs designed to meet the needs of 
disadvantaged children and apprizes the public and the legislature of 
program outcomes. In keeping with USOE requirements for evaluating 
Title I programs, this document is constructed of (1) responses to 
USOE probes by questionnaire sequence, (2) applicable supplementary 
or background information, and (3) available related findings. Data 
were collected from interviews with selected personnel from the 
Tennessee State Department of Education; reaction reports from 
teachers, administrators. State ESEA Title I personnel, and 
university personnel; onsite visitations by Title I staff and 
university consultants; and evaluation supplement and narrative 
reports distributed to local educational agency Title I directors and 
activity directors. Related documents are EA 003 692 and EA 003 721. 
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Introduction 



This report fulfills the obligation of the Tennessee State 
Department of Education to file an annual evaluation report with 
the United States Office of Education. The report is divided into 
five major parts. They are (1) Educational Achievement, (2) Changes 
in Administrative Structure and Educational Practices, (3) Coordination 
with other Federally Funded Programs, (4) Inservice Training, (5) Parent 
and Community Involvement. 

During FY'68 the Tennessee State Department of Education contract** 
ed with a state university to develop an evaluation design. The design 
was to have utility with projects or programs at the local school or 
school system level and with state programs. 

This approach to evaluation was initiated in order to (i) provide 
a planning mechanism compatible with a management systems approach, and 
(2) to provide a basis for training existing personnel in program 
evaluation since the number of persons with skills in evaluation is 
limited. 

The evaluation design is now being used on a field trial basis 
with ESEA Title III projects. Early indicators of its usefulness are 
promising. However, initial experiences with the design indicated a 
need for modification before wide application is affected. These 
modifications are being completed at the present time. A tentative 
outline follows t 
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A Tentative Outline 
of the 

Tennessee Planning and Evaluation Design 



Phase I 



Ip Status Study 

A. State educational goals 

B. Describe current status 

1. Define area of operation 
2 0 Describe the environment 

a. Community 

b. School 

3 o Gather readily attainable information 

a. Pupil personnel records 

b. Personnel records 
Cp Curriculum 

d. Out-of-school data 
4. Search for additional data 

Cp Organize data file 

Dp Analyze information 

1. Identify educational needs 
2 0 Determine underlying causes 

3. Identify educational assets 

Phase 2 

II * Program Planning 

A. Determine preliminary objectives directed toward 
identified need and general educational goals 
I* Who will be affected (population) 

2, Way they will be affected (behavior) 

3p Treatment that brings about change (program) 

4, Time interval 

Bp Develop program 

1. Examine alternative solutions for meeting objectives 

2. Consider constraints and resources for each solution 

3. Select most appropriate solution 

4p Select appropriate program activities for solution 
5o Develop supportive services 



4 



6 0 Develop operational pattern or organizational 
structure 

a. Define responsibilities 

b. Clarify lines of communication 

7. Assess capabilities for carrying out planned program 
and make necessary adjustments 

C* Develop procedural plan for assessment-an organized 
plan for providing information which will help guide 
future educational programs 

1* Consider various appropriate assessment techniques 

2. Select most appropriate technique; consider whether 
or not modifications should be made 

3. Determine general means of measurement 
a a Types of measurement devices needed 
bo Tentative selection 

4. Develop judgmental criteria«prof iciency level in 
terms of project 

5. Assess capabilities for carrying out selected 
assessment technique make adjustments as needed 

Do Develop calendar of events as an aid in planning and 
scheduling all major events in advance 
1. The major activities that will take place 
2o The date that these activities are expected to 
take place 

3, The materials and facilities which will be needed 
4* The personnel responsible for carrying out the 

activities 

5. The date the activity is completed 

Eo Develop monitoring system, a tool to see that the 

program is being implemented and conducted as planned 
I, Provide accurate picture of implementation of 
projects 

2o Allow for periodic checks on progress toward 
objectives 

3 0 Reveal strengths and/or weaknesses in proposed 
plan 

4. Provide the decision-maker with the information 
on which to base judgments 

5* Designate those people responsible for implementing 
the monitoring system 



Phase 3 



III. Program Operation 

A. Implement planned program 

B. State specific ob jectives-further refinement of 
preliminary objectives 

1 0 Population-those who are affected 

2. Program- the treatment or program innovation 

3. Behavior-the desired change in behavior 

4. Measurement technique-the type of measurement 
used to assess the behavior change 

5. Proficiency level-how well the learner must 
perform to reach the desired behavior. 

6. Time interval-the time needed to achieve an 
objective 

C. Select and/or develop measurement instruments- 
directed toward the specific objectives 

D. Implement monitoring system 

1. Make periodic observations 
2 0 Record descriptive data 

3. Use measurement instruments as scheduled 

E 9 Maintain record of events and activities 
1 0 Check against calendar of events 

2. Give descriptive history of project as it 
develops 

3. Point up strengths and weaknesses in planned 
calendar of events 

F. Analyze available data for program modification 

G. Identify program design defects and assessment 
technique defects 

Ho Modify program or assessment technique as needed 

Phase 4 

IV. Program Outcome 

A„ Complete data collection for total program 

Bo Analyze data concerning pre-post status and 
change measures 
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Co Interpret data in terms of judgmental criteria 

Do Formulate recommendations for future action such 

as re-cycling, expansion, modification, or termination 

E. Disseminate results as planned; other facets of 

dissemination may be built in at appropriate points 
in the program 

The revised design will be published before the beginning of FY'72. 

Workshops of short duration were conducted by Department staff 
for local school system ESEA Title I directors during the spring of 
197 Op These workshops focused on the development of behavioral 
objectives 0 

During the spring of 1971 conferences will be conducted for 
local school system superintendents and supervisors ■ A major 
objective of these sessions is to give conceptual visibility to 
the planning and evaluation design* 

In* depth workshops on the design and evaluation technique 
for local school system personnel with program assessment 
responsibilities are planned for the spring and summer of 1 97 1 o 
Television tapes on ESEA Title I evaluation are being developed 
utilizing the planning and evaluation design 0 These tapes will 
be used for local school system inservice education. 

Field trials of the modified design will be conducted 
during FY'72* If the field trials indicate the design is effective, 
wide application is expected during FY'73. 
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During the development of the planning and evaluation design, 
the Tennessee State Department of Education has monitored programs, 
initiated a needs assessment and gathered information that would 
yield gross indicators of the impact of various education projects 
and programs utilizing federal funds. Material included in this 
report is presented within this context. 

Educational Achievement 

Demonstrating the effectiveness of Title I projects by 
standardized achievement tests presents serious measurement and 
evaluation problems. The diversity of objectives, interventions, 
and evaluation strategies among projects forms the major barriers 
to generating meaningful state-wide summary data. These problems 
are compounded by internal problems such as (1) weaknesses in 
available instruments to measure school-effected change, (2) 
selecting suitable control groups and equating scores of 
different tests. 

In spite of these problems, the Tennessee State Department 
of Education drew together compilable test data from the LEA 
reports Q Following are some summary statements and gross, but 
sound, interpretations of the data c 

1 0 One hundred and forty-five project reports indicate 

movement toward objectives either by process evaluation 
or by product evaluation. 
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2. Sixty-eight of the 145 projects contain achievement 
test data. 

3. A few LEA's devised comprehensive evaluation strategies 
with built-in statistical analysis of test data to show 
that the expenditure of Title I funds did make a 
significant difference . 

4c Even though many LEA reports did not include a 

statistical analysis of test scores, and the gain in 
comparison with pre-test data or with national norms 
is slight, none of the reports indicates that Title I 
funds did not make a difference. 

5. Seventy of the evaluation reports are for Title I 
supported kindergartens. Twenty-three reports in- 
clude test data to show progress. 

6 0 Data pulled from a sample of kindergarten reports, 
which may or may not be representative of the total 
number, show that roughly seventy-five percent of 
the pupils had moved to a readiness-to-read status 
as defined in the Metropolitan Readiness Test Manual. 

7, Forty-five of the projects, representing 27,348 pupils, 
contain achievement test data which may or may not be 
representative of the total number of pupils in the 
147 Title I projectSo A gross summary of this data 
gives a .9 grade equivalent gain for the 27,348 pupils 
in all grade levels combined. The .9 grade equivalent 
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gain suggests that the typical project pupil gained 
almost as much during one year as the typical student 
in the national sample* 

Ch anges in Administrative Structure and Educational Practices 

Local school systems were asked to include in their evaluation 
reports a response to the questions: Has your Title I program affected 

the administrative structure of your agency, and have educational 
practices in your system been changed as a result of changes in the 
administrative structure? 

One hundred and forty~one local school systems indicated 
that Title I programs had affected their administrative structure 
and that changes in educational practices had resulted* Four 
reported no changes had resulted from Title I programs. 

The following items reported by local school systems are 
indicative of changes in administrative structure and educational 
practices * 

lo Administrative Structure 

a c The addition of administrative, supervisory and 
clerical staff to implement Title I programs 
the necessity of designing Title I programs as 
an integral part of a comprehensive compensatory 
educational program and the obligation to consider 
the relationship of the Title I program to the 
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regular school program have resulted in a significant 
increase in the involvement of staff and individual 
teachers in policy formation, curriculum planning, 
and administrative decisions. 

b. The functions of the advisory committees in the 
determination or priority needs has resulted in a 
significantly broader acceptance by boards of 
education of recommendations by lay groups. 

c. The design of programs to meet priority needs 
of educationally deprived children reflects a 
much wider practice of in-dgpth planning by 
principals, teachers, parents, and lay groups. 

d. The necessity to disseminate information relative 
to Title I has resulted in a meaningful exchange 
of information between local school systems. The 
more rural or isolated school districts reported 
this exchange of information to be especially 
helpful to them in program planning and evaluation. 

e. Approximately 90 percent of the local school systems 
reports indicated that the children of administrative/ 
supervisory staff had resulted in significant increases 
in coordination between the organizational divisions 

of their system. This coordination of the total 
activities of the administrative and instructional 
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divisions were most significant in curriculum develop- 
ment, attendance, pupil personnel services, and inservice 
training programs for both professional and nonprofessional 
personnel, 

2. Educational Practices 

a. Each of the local school systems reported a significant 
increase in individualized instruction. The increases 
reported generally evolved from: 

(1.) The assignment to the eligible centers of teacher 
aides and other nonprofessional personnel. 

(2.) Significant increases in instructional materials, 
supplies, and equipment. 

(3.) Project/program development design requiring 

assessment of needs and assignment of priorities 
and the development of measurable objectives 
related to needs. 

b. An estimated 75 percent of the systems reported 
significant increases in the awareness of a need 
for accountability. 

c. Fifty percent of the local systems reported that the 
use of consultants in preschool teacher training 
and in inservice education sessions had resulted in 
significant changes in teacher attitudes toward 
research and evaluation. 
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3. Coordination with other Federally Funded Programs 



The descriptions of the coordination of Title I programs 
with other federally funded programs indicated that while there 
was a generally high level of awareness of the nature and content 
of other federally funded programs, the resources of these agencies 
were not used at a level commensurate with this awareness, 

1* Each of the reporting local school systems indicated 
full use of the national school lunch program and 
its related programs such as special milk programs, 

2. Less than fifty percent of the local school systems 
reported the coordinated use of other federally 
funded programs in meeting high priority needs, 

3. Many local school systems reported either directly 
or by inference, the need for help in designing an 
effective, coordinated compensatory program using 
all available resources. 

Inservice Training 

Each reporting local school system recognized a responsibility 
to respond to, and reported on, the activities related to regulations 
requiring inservice training programs. Inservice is to be specifically 
directed toward Title I programs and the needs of Title I staff, and 
should allow the professional staff and the aides to participate 
together in the program. 
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1. One hundred of the local school systems reported the inservice 
training to have been a part of the preschool inservice train- 
ing usually held during the week prior to the beginning of 
the school year. 

2. Forty systems indicated, without detail, that inservice 
training of Title I staff was designed and implemented 
during the project and that provisions were made for 
inservice training of teachers and aides together. 

3. As indicated in the description of educational practices 
about 40 percent of the local systems reported inservice 
training of Title I staff, both professional and non- 
professional, as an entity rather than a component of 
the regular inservice training. 

4. Thirty local school systems reported that inservice 
training of Title I staff was a part of the required 
ten days of inservice training funded from state and 
local sources. 

Parent and Community Involvement 

State and federal guidelines/program guides give detail of 
the requirement of local advisory committees and of the necessity 
for parent and community involvement in program planning and pro- 
ject implementation. 
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All of the reporting systems described the nature and extent 
of the parent-community involvement „ The extent of involvement re- 
ported ranged from little involvement reported by 10 percent of the 
systems to significant involvement in 25 percent of the systems. 

1* Eighty percent of the 145 systems reporting on the 
activities of nine advisory committees reported that 
their contribution to project planning, development , 
and implementation generally ended with project planning 
and development and did not carry over into the implementation 
of project activities* 

2. Visits to schools and observation of project activities 
by parents were reported as the major activity in which 
parents were involved. Local systems reported a 
discernable increase in understanding by teachers of 
the implication of environmental factors in behavioral 
patterns and in the learning processes, 

3* Less than 10 percent of the local systems reported a 
significant level of volunteer work by parents or 
other members of the community. 

4. The most consistent participation by organized systems 
was that of the P 0 T 0 A. 

5* Supervision of recreation* transportation for field 
trips, transportation to clinics, and caring for in- 
structional supplies, helping to organize and transport 
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children to centers for medical, dental and optical 
examinations were activities most frequently identified 
as those in which volunteer parents participated. 

6. Relatively few local systems reported that parents of 
children participating in programs funded by ESEA 
Title I were employed as teacher aides or other non- 
professional personnel. 



16 



